In this video, we present a novel point-based, differentiable neural rendering pipeline.
From a set of input views and an initial 3D reconstruction, our system is able to synthesize
novel views and refine the scene's parameters.
Our pipeline consists of three major stages.
Rasterization, neural rendering, and tone mapping.
The rasterizer projects the point cloud to a given viewpoint and blends the assigned
neural descriptors into a multiscale output image.
Points that face away from the camera or fail a fuzzy depth test are discarded.
A deep neural network converts the multiscale input to a single HDR image.
The main tasks of the network are hole filling and shading.
The intermediate HDR image is passed to a differentiable tone mapper, which generates
the final LDR output.
During training, the physical parameters of the tone mapper are estimated.
For example, the camera response function and the per image exposure value.
In this experiment, we fit our pipeline to a data set of 688 images and 73 million points.
After a few iterations, we allow our system to also optimize the camera pose of each frame.
This further improves the sharpness of the rendering because the frames are now pixel
perfectly aligned.
Note here that our system is the first inverse rendering approach that can directly process
images of fisheye cameras.
We therefore can eliminate a potential lossy undistortion operation.
Here you can see a virtual flight over the playground scene from the tanks and temples
dataset.
This was generated by first fitting our pipeline to the dataset consisting of around 300 images
and 8 million points.
After that, we can synthesize novel views from arbitrary camera locations.
The same technique has been used to generate this video of a tank.
Here you can see the neural rendering of a rebuilt Roman vessel.
As you can see in the bottom right, the input images were captured with different exposure
settings.
Our system is still able to generate consistent novel views due to a physically based tone
mapper.
This also allows us to change the learned parameters of the tone mapper at inference
time.
Here you can see how we change the exposure time at a fixed camera location.
Thanks for watching.
Presenters
Zugänglich über
Offener Zugang
Dauer
00:03:45 Min
Aufnahmedatum
2021-11-17
Hochgeladen am
2021-11-17 13:26:19
Sprache
en-US
Paper on arXiv: https://arxiv.org/abs/2110.06635
Abstract: We present a novel point-based, differentiable neural rendering pipeline for scene refinement and novel view synthesis. The input are an initial estimate of the point cloud and the camera parameters. The output are synthesized images from arbitrary camera poses. The point cloud rendering is performed by a differentiable renderer using multi-resolution one-pixel point rasterization. Spatial gradients of the discrete rasterization are approximated by the novel concept of ghost geometry. After rendering, the neural image pyramid is passed through a deep neural network for shading calculations and hole-filling. A differentiable, physically-based tonemapper then converts the intermediate output to the target image. Since all stages of the pipeline are differentiable, we optimize all of the scene's parameters i.e. camera model, camera pose, point position, point color, environment map, rendering network weights, vignetting, camera response function, per image exposure, and per image white balance. We show that our system is able to synthesize sharper and more consistent novel views than existing approaches because the initial reconstruction is refined during training. The efficient one-pixel point rasterization allows us to use arbitrary camera models and display scenes with well over 100M points in real time.
Code on github: https://github.com/darglein/ADOP
More material on Zenodo: https://zenodo.org/record/5602606